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Executive  Summary 


Interoperability  is  the  ability  of  systems,  units,  or  forces  to  provide  services  to  and  accept 
services  from  other  systems,  units,  or  forces  and  to  use  the  services  exchanged  to  enable  them 
to  operate  effectively  together.  The  intent  of  this  paper  is  to  identify  current  practices  related 
to  measuring  systems  interoperability  and  to  recommend  a  set  of  measures  that  will  assist 
military  planners  in  the  acquisition,  development,  and  implementation  of  command,  control, 
communications,  computers,  and  intelligence  (C4I)  systems  that  are  interoperable. 

In  an  April  1998  report  to  Congress,  the  Secretary  of  Defense  noted  that  “joint  operations 
have  been  hindered  by  the  inability  of  forces  to  share  critical  information  at  the  rate  and  at 
the  locations  demanded  by  modern  warfare”  [Hamilton  00] .  Serious  interoperability 
deficiencies  have  been  perpetuated  across  all  the  services  and  have  been  identified  in  all 
recent,  allied,  joint,  and  combined  operations  and  exercises. 

Interoperability  is  a  broad  and  complex  subject.  Developing  and  applying  precise 
measurements  in  an  area  as  multidimensional  and  complex  as  interoperability  is  difficult. 
However,  measuring,  assessing,  and  reporting  interoperability  in  a  visible  way  is  essential  to 
setting  the  right  priorities.  An  increasing  importance  of  and  reliance  on  C4I  support  of 
military  operations  suggests  the  state  and  health  of  C4I  interoperability  be  characterized,  as 
much  as  possible,  in  a  more  explicit,  objective,  and  measurable  way. 

This  technical  note  reviews  the  state  of  the  practice  in  interoperability.  The  Levels  of  Systems 
Interoperability  (LISI)  Model  is  described.  This  model,  although  immature,  provides  a 
structured  and  systematic  approach  for  assessing  and  measuring  interoperability  throughout 
the  system  life  cycle.  A  summary  of  recommended  measures  that  could  promote  systems 
interoperability  in  the  Department  of  Defense  (DoD)  is  also  presented. 
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Abstract 


Despite  laudable  case-by-case  efforts,  there  is  today  no  method  for  tracking  interoperability 
on  a  comprehensive  or  systematic  basis.  This  technical  note  presents  best  practices  for 
measuring  systems  interoperability  and  assisting  military  planners  in  the  acquisition, 
development,  and  implementation  of  command,  control,  communications,  computers,  and 
intelligence  (C4I)  systems  that  are  interoperable.  The  Levels  of  Systems  Interoperability 
(LISI)  Model,  although  immature,  provides  a  structured  and  systematic  approach  for 
assessing  and  measuring  interoperability  throughout  the  systems  life  cycle.  In  addition  to 
exploring  the  many  complex  issues  surrounding  the  state  of  interoperability  for  military 
applications,  next  steps  for  promoting  a  deeper  understanding  of  interoperability  and 
recommended  measures  that  will  promote  systems  interoperability  are  presented. 
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1  Interoperability 


1.1  Introduction 

Command,  control,  communications,  computers,  and  intelligence  (C4I)  systems  relay  critical 
information  to  U.S.  forces  during  joint  operations. 

In  an  April  1998  report  to  Congress,  the  Secretary  of  Defense  noted  that  “joint  operations 
have  been  hindered  by  the  inability  of  forces  to  share  critical  information  at  the  rate  and  at 
the  locations  demanded  by  modern  warfare”  [Hamilton  00]. 

Parts  of  the  Department  of  Defense  (DoD)  are  well  aware  of  a  defense-wide  problem  in 
exploiting  rapidly  changing  information  technologies.  A  DoD  strategy  is  in  place  to  promote 
interoperability,  resting  on  technical  standards  such  as  the  Joint  Technical  Architecture  (JTA) 
and  use  of  a  defense -wide  common  infrastructure.  While  much  has  been  accomplished,  the 
goal  of  a  C4I  system  of  systems  with  interoperability  for  the  U.S.  military  continues  to  be 
unachieved.  Despite  increased  attention  and  management  awareness,  much  more  must  be 
done  before  C4I  interoperability  provides  adequate  end-to-end  support  of  military  missions. 

A  popular  perception  is  that  interoperability  is  synonymous  with  connectivity.  However,  true 
interoperability  is  much  more  than  just  connectivity.  It  is  also  a  function  of  operational 
concepts  and  scenarios,  policies,  processes,  and  procedures.  For  this  reason,  developing  and 
applying  precise  measurements  in  an  area  as  multidimensional  and  complex  as 
interoperability  is  difficult.  Interoperability  is  often  considered  to  be  a  desired  but 
unattainable  goal  rather  than  a  condition  that  can  be  quantified  [Leite  98].  Serious 
interoperability  deficiencies  exist  today.  They  have  been  perpetuated  across  all  the  services 
and  have  been  identified  in  all  recent,  allied,  joint,  and  combined  operations  and  exercises. 
Measurement  and  assessment — and  the  reporting  of  results  in  a  visible  way — are  essential  to 
setting  the  right  priorities.  As  noted  by  Presson  21  years  ago,  “interoperability  will  never  be 
an  analytically  useful  field  of  study  until  it  is  defined  in  a  quantitative  way”  [Presson  83]. 
Despite  laudable  case-by-case  efforts,  there  is  today  no  method  for  tracking  interoperability 
on  a  comprehensive  or  systematic  basis  [Committee  99]. 


CMU/SEI-2004-TN-003 


1 


1.2  Some  Definitions  of  Interoperability 

A  number  of  reports  and  technical  papers  offer  definitions  of  interoperability. 1  Recently,  a 
Joint  Chiefs  of  Staff  publication  [DoD  98]  defined  interoperability  to  acknowledge  both 
technical  and  operational  components. 

Table  1:  Defining  Interoperability 

Operational  Interoperability  The  ability  of  systems,  units,  or  forces  to  provide  services  to 

and  accept  services  from  other  systems,  units,  or  forces  and 
to  use  the  services  so  exchanged  to  enable  them  to  operate 
effectively  together  [DoD  95,  DoD  98]. 

The  ability  of  systems,  units,  or  forces  to  provide  services  to 
or  access  services  from  other  systems,  units,  or  forces,  and 
use  the  services  to  operate  effectively  together  [DoD  96]. 

Technical  Interoperability  The  condition  achieved  among  communications-electronics 

systems  or  items  of  communications-electronics  equipment 
when  information  or  services  can  be  exchanged  directly  and 
satisfactorily  between  them  and/or  their  users.  The  degree  of 
interoperability  should  be  defined  when  referring  to  specific 
cases  [DoD  98]. 

Interoperability  is  the  ability  of  systems  to  provide  dynamic 
interactive  information  and  data  exchange  among  C4I  nodes 
for  planning,  coordination,  integration,  and  execution  of 
Theater  Air  Missile  Defense  operations  [JTAMDO  97]. 

Operational  interoperability  addresses  support  to  military  operations  and,  as  such,  goes 
beyond  systems  to  include  people  and  procedures,  interacting  on  an  end-to-end  basis. 
Implementation  of  operational  interoperability  therefore  implies  both  the  traditional  approach 
of  defining  standards  as  well  as  enabling  and  assuring  activities  such  as  testing  and 
certification,  configuration,  and  version  management,  and  training  [Committee  99]. 

Interoperability  at  the  technical  level  is  essential  to  achieving  operational  interoperability. 
Technical  interoperability  occurs  between  systems  (as  opposed  to  organizations)  and  should 
be  considered  in  a  variety  of  contexts  and  scopes.  Dimensions  of  technical  interoperability 
include 

•  sensors  generating  bits  of  information 

1  More  information  on  this  topic  is  available  in  Appendix  A. 
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•  communication  channels  transmitting  the  bits  of  information 

•  computers  processing  the  bits  of  information 

•  weapons  directed  by  messages  composed  of  bits 

For  two  C4I  systems  to  effectively  interoperate,  they  must  be  able  to  exchange  relevant 
bitstreams  as  well  as  to  interpret  the  bits  they  exchange  according  to  consistent  definitions. 
Thus,  technical  interoperability  places  detailed  demands  at  multiple  levels,  which  range  from 
physical  interconnection  to  correct  interpretation  by  applications  of  data  provided  by  other 
applications  [Committee  99], 
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2  An  Architectural  Approach  to  Technical 
Interoperability 


The  architecture  of  a  system  is  the  structure  or  structures  of  the  system  that  comprise  the 
components,  the  externally  visible  properties  of  those  components,  and  the  relationships 
among  them  [Bass  98],  Using  measurement  to  assess  the  behavior  of  the  key  attributes  of 
these  components  (and  the  relationships  between  them)  is  a  daunting  task.  An  architectural 
perspective  helps  to  organize  the  complexity  of  the  interoperability  challenge  in  ways  that 
can  lead  to  more  coherent  treatments. 

Architectures  are  a  hierarchical  description  for  the  design  of  a  system  and  in  many  cases 
describe  how  it  will  be  developed,  evolved,  and  operated.  Architectures  provide  the 
underlying  blueprint  for  the  more  detailed  design  and  implementation  decisions  about  the 
components  of  a  system.  When  well-defined  architectures  exist,  engineers  can  design 
individual  components  and  builders  can  implement  them  with  a  high  degree  of  confidence 
that  the  end  results  will  work  as  expected  and  meet  user  needs. 

There  are  a  number  of  architectural  characteristics  that  can  be  used  as  a  basis  for  reasoning 
about  what  might  be  considered  appropriate  quality  attributes  that  can  be  measured.  These 
include  interfaces  and  layers,  standards,  and  data  interoperability. 

When  reasoning  about  architecture,  it  makes  sense  to  strive  for  an  information-systems 
environment  based  on  well-defined  requirements  specification,  common  data  structures, 
common  interface  requirements,  and  well-specified  high-level  information  flows.  Systems 
constructed  in  accordance  with  such  an  architecture  are  much  more  likely  to  be  adequately 
interoperable  than  those  that  are  not.  However,  particularly  when  legacy  systems  are 
involved,  these  commonalities  may  not  exist.  In  these  cases,  architectures  can  supply 
valuable  guidance  to  isolate  gaps  and  risks  relative  to  interoperability. 


2.1  Interfaces  and  Layers 

The  modular  decomposition  of  systems  is  typically  both  horizontal  and  vertical.  Vertical 
decomposition  refers  to  interfaces  between  discrete  systems  within  the  same  layer  (e.g.,  a 
standard  message  format  used  by  different  applications  to  exchange  information).  Horizontal 
decomposition  of  functions  is  known  as  layering  (e.g.,  the  separation  of  bit  transport 
technologies,  transport  protocol,  and  applications). 
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2.1 .1  Interfaces 

Systems  that  perform  a  variety  of  functions  are  normally  composed  of  multiple  subsystems  or 
components.  Interfaces  arise  whenever  one  subsystem  or  component  interacts  with  another. 
An  architect  that  is  designing  and  partitioning  a  system  that  is  intended  to  interoperate  with 
unspecified  existing  and  future  systems  must  consider  the  importance  of  the  following: 

•  Interface  design.  Well-designed  and  documented  interfaces  that  permit  development 
programs  to  be  divided  into  more  manageable  pieces.  This  results  in  faster  development 
because  the  work  of  different  players  can  proceed  in  parallel. 

•  Encapsulation.  This  permits  modular  change  in  version  and  implementation  technology. 
By  encapsulating  the  internal  details  of  a  system  component  which  may  change  over 
time,  interfaces  allow  changes  in  internal  implementation  of  portions  of  a  system  to  be 
transparent  to  other  external  systems. 

•  Reducing  interaction.  Reducing  the  complexity  of  intersystem  dependencies  facilitates 
more  rapid  reconfiguration  of  systems  to  meet  operational  requirements. 

2.1 .2  Layers 

Layers  facilitate  making  C4I  systems  interoperable  in  the  presence  of  rapidly  changing 
technologies  and/or  multiple  technology  choices.  Layering  makes  it  possible  to  design  a 
system  of  systems  that  has  technology  independence,  scalability,  decentralized  operation, 
appropriate  architecture,  and  supporting  standards,  security,  and  flexibility.  Layering  can  also 
accommodate  heterogeneity,  accounting,  and  cost  recovery  [CSTB  94],  Excellent  examples 
of  layering  include  the  use  of  TCP/IP  to  decouple  communications  link  technologies  from 
applications  that  use  communications  and  the  use  of  hypertext  transport  protocol  (HTTP)  and 
hypertext  markup  language  (HTML)  to  separate  presentation  from  storage  and  retrieval 
functions. 

Middleware  provides  an  example  of  the  layering  principle.  It  separates  the  applications  from 
the  operating  systems  on  which  the  applications  run  [SEI  00],  [Bernstein  96].  As  outlined  in 
Ligure  1 ,  Middleware  services  are  sets  of  distributed  software  that  exist  between  the 
application  and  the  operating  system  and  network  services  on  a  system  node  in  the  network. 

By  decreasing  the  dependence  of  applications  on  a  particular  operating  system,  middleware 
increases  the  ease  of  moving  applications  to  new  computers  or  systems  and  decreases 
dependence  on  operating  systems  that  might  fall  out  of  favor  in  the  commercial  marketplace. 
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Figure  1:  Middleware 

The  Common  Object  Request  Broker  Architecture  (CORBA)  is  one  of  a  number  of 
Middleware  platforms  for  distributed  systems  development  [OMG  98].  Other  platforms 
include  Microsoft’s  .NET  and  Java  2  Platform,  Enterprise  Edition.  CORBA  specifies  a 
system  that  provides  interoperability  among  objects  in  a  heterogeneous,  distributed 
environment  and  in  a  way  that  is  transparent  to  the  programmer.  It  enables  applications  to 
cross  the  boundaries  of  different  computing  machines,  operating  systems,  and  programming 
languages.  It  specifies  how  client  applications  can  invoke  operations  on  server  objects. 
Abundant  information  about  CORBA  is  available  from  various  resources  [Cetus  00]. 


2.2  Standards 

An  essential  aspect  of  architecture  is  the  establishment  of  technical  standards.  In  general, 
standards  define  common  elements,  such  as  user  interfaces,  system  interfaces,  representations 
of  data,  protocols  for  the  exchange  of  data,  and  interfaces  accessing  data  or  system  functions. 

Technical  standards  provide  a  number  of  advantages  for  the  systems  architect.  With  regard  to 
interoperability,  standards  are  important  because  they  are  accepted  by  multiple  vendors, 
thereby  increasing  the  likelihood  that  a  collection  of  systems  from  diverse  sources  will  be 
able  to  interoperate.  It  has  become  generally  accepted  by  now  that  although  standards  are 
certainly  beneficial,  simple  adherence  to  standards  is  not  sufficient  to  guarantee 
interoperability  [NIMA  98].  Even  when  there  are  accepted  standards  and  compliant  products, 
interoperability  is  facilitated  but  not  assured  as  there  are  options  within  standards  and 
different  releases  and  versions  of  products. 

Finally,  it  is  important  to  realize  that  technical  standards  are,  by  themselves,  necessarily 
incomplete  from  the  standpoint  of  a  system  or  component  designer.  The  operational  scenarios 
that  a  system  is  expected  to  support  play  an  integral  role.  This  range  of  scenarios  defines  the 
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context  in  which  a  system  is  to  perform  specific  desired  functions  and  thus  provides  a 
meaningful  reference  for  testing  and  evaluation. 


2.3  Data  Interoperability 

Experience  suggests  that  left  to  their  own  devices,  the  designers  of  individual  systems  will 
often  make  locally  optimal  decisions  about  data  definitions  and  formats  [Committee  99]. 

Data  formats  resulting  from  such  local  decisions  may  not  be  compatible  when  operational 
requirements  dictate  that  a  network  of  systems  be  called  upon  to  interoperate.  Thus 
architectural  design  must  provide  guidance  to  developers  to  minimize  the  applications-layer 
incompatibilities  that  inevitably  arise  when  systems  with  different  purposes  must 
communicate  with  each  other. 

Examples  of  approaches  to  data  interoperability  include 

•  Single  data  definition  for  all  systems.  This  approach  can  be  problematic  when  applied  on 
a  large  scale  to  a  complex,  evolving  system  or  system  of  systems.  The  task  of  agreeing  on 
definitions  consumes  a  great  deal  of  effort  and  time  that  might  be  better  used  elsewhere. 
Also,  when  a  single  set  of  definitions  is  mandated  for  all  applications,  definitions  are  no 
longer  locally  optimal,  and  thus  such  mandates  often  encounter  substantial  resistance  in 
implementation. 

•  Object  orientation.  This  is  a  technically  promising  approach  for  developing  data 
definitions  by  encapsulating  the  internal  details  of  the  data  [Committee  99]. 

•  Extensible  data  model.  This  approach  uses  an  extensible  data  model  and  standardized 
interface.  The  Simple  Network  Management  Protocol  is  an  example  [Cherkaoui  99]. 

•  Extensible  Markup  Language  (XML).  This  approach  requires  agreement  on  the  contents 
and  meaning  of  the  XML  schema.  Thereafter,  application  data  is  communicated  in  XML 
that  conforms  to  the  schema.  Like  single  data  definition,  obtaining  agreement  on  the 
schema  can  be  difficult  but  XML  also  promises  extensibility  of  data  markup. 

Legacy  systems  which  have  been  built  around  frequently  unique  data  definitions  pose  a  major 
challenge  to  interoperability.  Industry  has  developed  a  number  of  approaches  by  which 
systems  not  originally  designed  for  interoperability  can  interoperate  to  exchange  information 
(including  the  data  “bus”  approach,  the  data  dictionary  approach,  the  data  translator 
approach,  and  the  data  server  approach). 


2.4  DoD  Strategy  for  Addressing  Interoperability 

In  recognition  of  the  importance  of  interoperability  to  realizing  its  C4I  goals  (Joint  Vision 
2010  [Chairman  96]  and  Joint  Vision  2020  [Chairman  00]),  the  DoD  has  adopted  a 
joint/defense -wide  strategy  for  promoting  interoperability. 
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Specifically,  there  are  three  major  elements  that  have  emerged 

1 .  a  triad  of  interrelated  architectures 

2.  a  common  defense -wide  infrastructure  with  a  common  applications  platform 

3.  applications-level  efforts  to  promote  interoperability 

The  three-part  architecture  is  conceptually  presented  in  Table  1  and  described  by  Chatfield 
[Chatfield  98],  It’s  important  to  note  that  the  architectures  are  not  all  at  the  same  level  of 
development.  DoD  architectural  development  to  date  has  focused  on  the  Joint  Technical 
Architecture  (JTA)  and  it  is  by  far  the  most  mature  architecture  of  the  three. 


Table  2:  Elements  of  the  DoD  Architectural  Triad  [C4ISR  97] 


Part  of  Triad 

Description 

Joint 

Operational 

Architecture 

A  description  (often  graphical)  of  the  operational  elements,  assigned  tasks,  and 
information  flows  required  to  support  the  warfighter.  It  defines  the  type  of 
information  exchanged,  the  frequency  of  the  exchange,  and  what  tasks  are 
supported  by  these  information  exchanges.  The  operational  architecture  is  a 
doctrine-driven  representation  of  C4ISR  nodes,  roles,  processes, 
interrelationships,  and  data/information  exchanges.  This  representation  relates 
to  specific  scenarios  and  joint/combined/coalition  mission  functions  and  forms 
the  basis  for  realistic  process  and  information  flow  representation  and 
prioritization. 

Joint 

Systems 

Architecture 

A  description,  including  graphics,  of  the  systems  and  interconnections 
providing  for  or  supporting  a  warfighting  function.  The  systems  architecture 
(view)  defines  the  physical  connection,  location,  and  identification  of  the  key 
nodes,  circuits,  networks,  warfighting  platforms,  etc.  associated  with 
information  exchange  and  specifies  system  performance  parameters.  The 
systems  architecture  (view)  is  constructed  to  satisfy  operational  architecture 
component  requirements  per  the  standards  defined  in  the  technical  architecture. 

Joint 

Technical 

Architecture 

A  minimal  set  of  rules  governing  the  arrangement,  interaction,  and 
interdependence  of  the  parts  or  elements  whose  puipose  is  to  ensure  that  a 
conformant  system  satisfies  a  specific  set  of  requirements.  It  identifies  system 
services,  interfaces,  standards,  and  their  relationships.  It  provides  the 
framework  upon  which  engineering  specifications  can  be  derived,  guiding  the 
implementation  of  systems. 
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Figure  2:  Three  Views  of  an  Architecture 

The  JTA  is  intended  to  provide  a  set  of  correct  and  mutually  consistent  technical  standards, 
application  interfaces  (APIs),  and  protocols,  along  with  the  decision  rules  for  using  them. 

The  Joint  Operational  Architecture  was  originally  intended  to  be  a  construct  covering  all 
military  operations.  The  Information  Superiority  Campaign  Plan  of  the  Joint  Chiefs  of  Staff 
calls  for  “the  development  of  a  high-level,  C4  Joint  Operational  Architecture  that  integrates 
the  joint  warfare  functions,  from  national  level  through  operational  level,  into 
implementations  of  the  JV2010  (Joint  Vision  2010)  operational  concepts”  [JCS  00]. 

An  additional  piece  of  the  DoD  strategy  for  C4I  interoperability  is  the  building  of  a  common, 
defense- wide  information  infrastructure  called  the  Defense  Information  Infrastructure  (DII). 
The  Dll  includes  a  set  of  common  software  called  the  Dll-Common  Operating  Environment 
(COE).  The  DII-COE  includes  increasingly  capable  middleware,  on  top  of  which 
service/mission-specific  applications  can  be  built.  Use  of  the  DII-COE  and  achieving 
compliance  at  certain  levels  is  specified  in  the  JTA. 

With  regard  to  data  interoperability,  the  DoD  understands  the  importance  of  data  integration 
and  has  launched  two  major  efforts  in  this  area: 

1 .  The  Enterprise  Data  Model  Initiative  [DoD  91] 

2.  Shared  Data  Environment  (SHADE)  program  [DISA  96] 
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The  Enterprise  Data  Model  Initiative  sets  forth  a  DoD  process  through  which  standard  data 
definitions  in  C4I  functional  areas  are  developed.  They  are  then  subjected  to  a  cross¬ 
functional  review  process  prior  to  being  adopted  as  DoD  standards.  The  goal  of  this  process 
is  to  develop  a  complete  set  of  standard  data  elements  for  DoD  applications. 

The  SHADE  program  relies  on  a  “bottom-up”  approach  to  enable  different  C4I  systems  to 
share  data  segments  and  to  use  standardized  access  methods.  SHADE  has  demonstrated  some 
success  in  enabling  legacy  systems  to  interoperate.  This  program  has  recently  been  subsumed 
by  DII-COE. 


10 
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3  Policies  Related  to  Interoperability 


Current  systems  are  increasingly  being  built  to  meet  explicit  requirements  for  interoperability 
and  flexibility.  The  DoD’s  vision  of  the  future — Joint  Vision  2010 — is  one  of  information 
superiority  [Chairman  96].  The  cornerstone  of  information  superiority  is  advanced  C4I 
technology  and  systems  which  can  provide  a  robust,  continuous,  common  operating  picture 
of  the  battlespace  to  all  tactical  levels  of  command.2  The  common  operating  picture  is  a 
central  element  in  a  number  of  initiatives,  including 

•  The  Army  Digitization  Master  Plan  (Force  XXI)  [Army  96] 

•  The  Theater  Air  and  Missile  Defense  Program  [TMD  Plan  98] 

•  The  Battlefield  Awareness  and  Data  Dissemination  (BADD)  advanced  concept 
technology  demonstration  (ACTD)  [OUSD  99a] 

•  The  “Extending  the  Littoral  Battlespace”  (ELB)  ACTD  [OUSD  99b] 

Joint  Vision  2010  and  Joint  Vision  2020  reflect  the  top-level  vision  in  the  DoD  of  what  is 
possible  though  the  exploitation  of  technology  to  solve  the  interoperability  problem 
[Chairman  96],  [Chairman  00].  Each  of  the  services  has  translated  this  top-level  vision  into  a 
service-specific  vision  [Dept,  of  Army  96],  [Dept,  of  Air  Force  96],  [Dept,  of  Navy  96], 
[Marine  96].  Each  service  is  exploring  the  implications  of  Joint  2010  and  Joint  2020,  taking 
steps  with  experimental  studies,  wargames,  research  and  development  activities,  and 
simulation  gaming  to  develop  and  test  concepts  and  capabilities  that  will  ensure  military 
preparedness  for  the  coming  decades.  Additionally,  as  an  extension  of  individual  service 
experimentation,  and  in  response  to  congressional  pressures,  a  joint  experimentation  activity 
is  being  established  at  the  U.S.  Atlantic  Command  to  address  the  co-evolution  of  doctrines, 
tactics,  and  new  technological  capabilities  [Committee  99]. 

The  DoD  has  a  number  of  other  initiatives  underway  that  address  various  aspects  of 
interoperability  including 

•  C4I  for  the  Warrior  Concept 

•  Command,  Control  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  Architecture  Framework 

•  Defense  Information  Infrastructure  Strategy 

•  Levels  of  Information  Systems  Interoperability  Initiative 


2 


The  term  “common  operating  picture”  refers  to  a  view  of  the  battlespace  that  is  near  real-time. 
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•  Global  Information  Grid  Architecture,  including  ForceNet,  C2  Constellation,  and 
LandWarNet 

All  of  these  initiatives  are  about  improving  the  interoperability  of  C4I  Systems  [GAO  98]. 


12 
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4  Information  Needs 


Historically,  DoD  approaches  to  interoperability  have  ranged  from  handling  it  on  a  program- 
by-program  basis  to  making  limited-scope  efforts  on  a  joint,  community-wide  basis  (e.g.,  the 
Joint  Interoperability  of  Tactical  Command  and  Control  Systems  activity  to  address  joint 
message  standards)  or  a  functional  community  basis  (e.g.,  air  defense).  In  addition,  some 
programs  to  develop  defense-wide  infrastructure,  dating  back  to  at  least  the  1960s,  have  been 
followed  more  recently  by  a  few  sizable,  centrally  managed  application  development 
programs  (e.g.,  the  Global  Command  and  Control  System  as  a  replacement  for  the  Worldwide 
Military  Command  and  Control  System  [Committee  99]). 

However,  the  responsibility  for  interoperability  is  now  distributed  across  the  DoD  and  each  of 
the  higher  ranks  of  command  has  at  least  one  entity  charged  with  responsibility  for 
interoperability  issues3  as  shown  in  the  table  below. 


Table  3:  Interoperability  Entities  for  Agencies  and  Commands  in  the  DoD 


Agency  or  Command 

Entity  Responsible  for  Interoperability 

U.S.  Atlantic  Command 

Joint  Battle  Center 

Joint  Staff 

Military  Communications  and  Electronics  Board 

Assistant  Secretary  of  Defense  for  C3I 

Information,  Integration,  and  Interoperability  Directorate 

Defense  Information  Systems  Agency 

Joint  Interoperability  Test  Command 

DoD  guidance  requires  that  a  system  be  tested  and  certified  before  approval  to  produce  and 
field  it.  Depending  on  the  acquisition  category  and  dollar  threshold  of  the  program,  the 
approval  authority  may  be  one  of  the  following  [GAO  98]: 

•  Undersecretary  of  Defense  (Acquisition  and  Technology),  with  advice  from  the  Defense 
Acquisition  Board 

•  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence), 
with  advice  from  the  Major  Automated  Information  System  Review  Council 


3 

The  listing  of  organizations  is  far  from  complete.  There  is  a  multiplicity  of  organizations  and  offices  with 
some  responsibility  for  C4I  matters,  and  organizational  structures  for  C4I  (in  general)  have  been  rapidly 
evolving. 
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•  DoD  component  head  (such  as  the  commander-in-chief  of  a  unified  combatant 
command,  the  head  of  a  military  service,  or  a  DoD  agency  head) 

To  ensure  interoperability,  the  Defense  Information  Systems  Agency  (DISA),  under  the 
direction  of  the  Joint  Chiefs  of  Staff,  established  the  current  C4I  interoperability  certification 
process  in  1992.  According  to  Joint  Staff  guidance,  commanders-in-chief,  the  four  services, 
and  DoD  agencies  are  required  to  use  this  process  to  test  and  certify  existing  and  newly 
developed  systems  for  interoperability. 

The  Joint  Staff’s  Director  for  C4  Systems  (J-6)  is  assigned  primary  responsibility  for 
ensuring  compliance  with  the  certification  requirements.  DISA’s  Joint  Interoperability  Test 
Command  is  the  sole  certifier  of  C4I  systems.  According  to  Joint  Staff  guidance, 
commanders-in-chief  (CINCs),  the  services,  and  DoD  agencies  are  required  to  adequately 
budget  for  certification  testing  [GAO  98]. 

When  thinking  about  these  stakeholders,  one  may  note  the  different  perspectives  that  are 
working  based  on  the  role  of  the  stakeholder.  Some  entities  are  operational  while  others  are 
more  focused  on  planning.  Operational  units  (in  the  DoD  context,  CINCs  are  the  warfighting 
authorities)  have  a  perspective  concerned  with  the  capabilities  of  today’s  systems  (in  the  short 
term).  Planning  units4  are  concerned  with  the  long-term  capabilities  of  tomorrow’s  systems. 


4  For  example,  the  Office  of  the  Secretary  of  Defense,  Joint  Chiefs  of  Staff,  and  the  service  chiefs  operate  as  the 
policy  makers,  allocators  of  resources,  and  acquirers. 
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5  Measures  for  Interoperability 


Interoperability  is  typically  assessed  by  the  DoD  through  non-comprehensive  perspectives 
that  are  focused,  for  example,  on  standards  (e.g.,  the  JTA),  Common  Operating  Environment 
(COE)  compliance,  data  models,  or  certification  criteria,  and  how  individual  systems 
compare  against  such  criteria  or  standards.  It  is  generally  recognized  that  much  more  needs  to 
be  accomplished  in  this  area  [Committee  99]. 

The  popular  perception  is  that  interoperability  is  synonymous  with  connectivity.  However,  as 
previously  mentioned,  true  interoperability  is  much  more  than  just  connectivity.  As  Robert  M. 
Nutwell,  deputy  secretary  of  defense  for  command,  control,  communications,  and 
intelligence,  surveillance,  and  reconnaissance  systems,  explained: 

Integration  is  generally  considered  to  go  beyond  mere  interoperability  to 
involve  some  degree  of  functional  dependence.  For  example,  a  mission 
planning  system  might  rely  on  an  external  intelligence  database;  an  air 
defense  missile  system  will  normally  rely  on  acquisition  radar.  While 
interoperable  systems  can  function  independently,  an  integrated  system  loses 
significant  functionality  if  the  flow  of  services  is  interrupted.  An  integrated 
family  of  systems  must  of  necessity  be  interoperable,  but  interoperable 
systems  need  not  be  integrated. 

Compatibility  is  something  less  than  interoperability.  It  means  that 
systems/units  do  not  interfere  with  each  other’s  functioning.  But  it  does  not 
imply  the  ability  to  exchange  services.  Interoperable  systems  are  by  necessity 
compatible,  but  the  converse  is  not  necessarily  true.  To  realize  the  power  of 
networking  through  robust  information  exchange,  we  must  go  beyond 
compatibility. 

In  sum,  interoperability  lies  in  the  middle  of  an  “Integration  Continuum” 
between  compatibility  and  full  integration.  It  is  important  to  distinguish 
between  these  fundamentally  different  concepts  of  compatibility, 
interoperability,  and  integration,  since  failure  to  do  so  sometimes  confuses 
the  debate  over  how  to  achieve  them.  While  compatibility  is  clearly  a 
minimum  requirement,  the  degree  of  interoperability/integration  desired  in  a 
joint  family  of  systems  or  units  is  driven  by  the  underlying  operational 
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concept,  as  well  as  by  family  of  systems  (FoS)  design  and  cost/effectiveness 
tradeoffs.5 

It  is  also  a  function  of  operational  concepts  and  scenarios,  policies,  processes,  and 
procedures.  For  this  reason,  developing  and  applying  precise  measurements  in  an  area  as 
multidimensional  and  complex  as  interoperability  is  difficult.  However,  the  increasing 
importance  of  and  reliance  on  C4I  support  of  military  operations  suggests  that  the  state  and 
health  of  C4I  interoperability  be  characterized  in  a  more  explicit,  objective,  and  measurable 
way. 

To  account  for  the  multi-faceted  nature  of  the  interoperability  domain,  we  propose  four  sets 
of  measures  that  address  the  following  aspects  of  this  challenging  problem  space: 

1 .  technical  compliance  measures 

2.  systems  interoperability  measures 

3.  operational  interoperability  measures 

4.  organizational  and  cultural  measures 

The  first  three  sets  of  measures  are  discussed  in  the  context  of  the  Levels  of  Systems 
Interoperability  (LISI)  Model.  This  evolving  model  is  described  in  Section  5.1,  and  Sections 
5.1.1  and  5.1.2  describe  proposed  approaches  for  addressing  the  measurement  areas  listed  in 
the  first  three  numbered  items  above. 

In  addition,  it  is  now  generally  accepted  that  management  must  be  able  to  measure  what  they 
wish  to  change.  Achieving  large-scale  cultural  change  (that  is  required  to  bring  about 
interoperability)  requires  commensurate  change  in  management  and  the  organizational 
measures  of  performance.  In  Section  5.2,  a  starter  set  of  important  management  measures  for 
assessing  progress  related  to  interoperability  is  recommended.  Section  5.3  describes  tradeoff 
considerations  that  must  be  factored  in  as  part  of  the  challenge  to  promote  systems 
interoperability. 


5.1  Levels  of  Information  Systems  Interoperability  (LISI) 

The  LISI  project  was  initiated  in  1993  by  MITRE,  the  C4ISR  Integration  Task  Force,  and  the 
ACC  Architecture  Working  Group  [C4ISR  98].  LISI  is  a  reference  model  and  process  for 
assessing  information  systems’  interoperability.  It  is  a  discipline  and  a  process  for  defining, 
measuring,  assessing,  and  certifying  the  degree  of  interoperability  required  or  achieved 
between  organizations  or  systems. 


5 

Robert  M.  Nutwell,  prepared  speech  on  “Achieving  Joint  Information  Interoperability,’"  Version  1,  April, 
2000. 
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LISI  assesses  the  level  of  interoperability  attained  between  systems  (not  between  users). 

Once  system-to-system  interoperability  issues  have  been  isolated,  the  ability  to  address  user 
interoperability  issues  is  vastly  improved.  For  example,  LISI  concentrates  on  which  user 
problems  are  related  to  functional  training  needs/shortfalls,  differing  operational  methods  and 
procedures,  and  difficulties  in  user-to-computer  interactions.  Figure  3  (adapted  from 
Hamilton  [Hamilton  00])  illustrates  this  capability. 


User: 
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Figure  3:  LISI  Scope  of  Analysis 

LISI  uses  a  common  frame  of  reference  and  measure  of  performance.  LISI  is  applied 
throughout  the  information  system  life  cycle,  from  requirements  analysis  through  systems 
development,  acquisition,  fielding,  and  subsequent  improvement  and  modification.  In  this 
context,  LISI 

•  facilitates  a  common  understanding  of  interoperability  and  the  suite  of  capabilities 
enabling  each  logical  level  of  system-to-system  interaction 

•  provides  an  interoperability  maturity  model  and  associated  requisite  capabilities  as  the 
basis  for  making  comparisons  between  heterogeneous  systems  and  maturing  individual 
systems 

•  provides  a  methodology  for  assessing  and  improving  interoperability  by  guiding 
requirements  and  architecture  analysis,  systems  development,  acquisition,  fielding,  and 
technology  insertion 


Table  3  presents  an  overview  of  the  LISI  Interoperability  Maturity  Model.  This  model 
identifies  the  stages  through  which  a  system  should  logically  progress  to  improve  its 
capabilities  to  interoperate.  LISI  considers  five  increasing  levels  of  sophistication  regarding 
system  interaction  and  the  ability  of  the  system  to  exchange  and  share  information  and 
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services.  Each  higher  level  represents  a  demonstrable  increase  in  capabilities  over  the 
previous  level  of  system-to-system  iteration. 


Table  4:  Overview  of  the  LISI  Interoperability  Maturity  Model 


Information  Exchange 


Level 


Computing  Environment 


Distributed  global  information  and 
applications 

Simultaneous  interactions  with  complex 
data 

Advanced  collaboration,  e.g.  interactive 
COP  update 


4 


Enterprise 


Interactive  manipulation; 
shared  data  and  applications 


Event-triggered  global  database  update 


Shared  databases  3 


Sophisticated  collaboration,  e.g.,  Domain 

Common  Operating  Picture 

Shared  data;  separate 
applications 


Warfighter  #1 


Warfighter  #2 


Warfighter  #3 


Battle  Manager 


Heterogeneous  product  exchange 

Basic  collaboration 

Group  collaboration,  e.g.,  exchange  of 
annotated  imagery,  maps  with  overlays 


2 

Functional 

Minimal  common 
functions;  separate  data  and 
applications 


Homogeneous  product  exchange,  e.g., 
FM  voice,  tactical  data  links,  text  files, 
transfers,  message,  e-mail 


1 

Connected 

Electronic  connection; 
separate  data  &  applications 


Manual  gateway,  e.g.,  diskette,  tape,  hard 
copy  exchange 


0 

Isolated 


Non-connected 
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A  critical  element  of  interoperability  assurance  is  a  clear  prescription  of  the  common  suite  of 
requisite  capabilities  that  must  be  inherent  in  all  information  systems  that  desire  to 
interoperate  at  a  selected  level  of  sophistication.  Each  level’s  prescription  of  capabilities  must 
cover  all  four  enabling  attributes  of  interoperability,  as  shown  in  the  table  below. 


Table  5:  PAID  Attributes 


□ 

Procedures 

Policies  and  procedures  govern  a  system' s  development  through  established  standards  and 
the  procedures  and  processes  which  influence  system  integration  and  functional 
operational  requirements 

□ 

Applications 

The  functions  a  system  is  intended  to  perform.  These  functions  reside  most  often  in  the 
form  of  user-based  application  programs  which  perform  or  support  a  specific  set  of 
processes  or  procedures. 

D 

Infrastructure 

The  infrastructure  required  to  support  the  systems  operations.  Contains  four  sub¬ 
components  which  are  also  defined  in  terms  of  increasing  levels  of  sophistication. 

□ 

Data 

The  data  and  information  structures  used  to  support  both  the  functional  applications  and 
system  infrastructure. 

In  addition,  for  each  prescribed  capability,  system  developers  need  to  know  what 
implementation  options  are  available,  and  which  options  conform  to  prevailing  DoD  criteria. 
The  LISI  Capabilities  Model  and  its  associated  Implementation  Options  Tables  identify  the 
full  suite  of  capabilities  and  available  technical  implementations  for  attaining  each  level  of 
interoperability.  Table  5  summarizes  the  LISI  Reference  Model  and  shows  the  relationship  of 
the  PAID  attributes. 
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Table  6:  LISI  Reference  Mode 


Interoperability  Attributes 

Description 

Computing  Environment 

Level 

Procedures 

Application 

Infrastructure 

Data 

Enterprise 

Universal 

4 

Enterprise 

Level 

Interactive 

Multiple 

Dimensional 

Topologies 

Enterprise 

Model 

Domain 

Integrated 

3 

Domain  Level 

Groupware 

Worldwide 

Network 

Domain 

Model 

Functional 

Distributed 

2 

Program  Level 

Desktop 

Automation 

Local 

Networks 

Program 

Model 

Connected 

Peer-to-Peer 

1 

Local/Site 

Level 

Standard 

System 

Drivers 

Simple 

Connection 

Local 

Isolated 

Manual 

0 

Access  Control 

N/A 

Independent 

Private 

Table  6  (adapted  from  a  1998  report  [C4ISR  98])  presents  a  general  overview  of  the  major 
elements  that  comprise  LISI.  LISI  provides  an  assessment  process  for  determining  the 
interoperability  maturity  level  or  “measure”  of  a  given  system  or  system  pair.  (Note  in  Table 
6  that  Interoperability  Metrics  is  included  as  one  of  the  LISI  assessment  products.) 
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LISI  Assessment  Products  LISI  Assessment  Basis 


Table  7:  Overview  of  the  LISI  Elements 


LISI  Element 

Description 

Interoperability  Maturity 
Model 

Defines  the  five  levels  of  interoperability  expressed  within  LISI. 

The  LISI  interoperability  Maturity  Model  describes  the  increasing 
sophistication  of  system-to-system  interactions  as  one  moves  from 
one  level  to  the  next. 

Reference  Model 

Characterizes  the  five  levels  of  interoperability  in  terms  of  four 
comprehensive,  integrated  attributes:  procedures,  applications, 
infrastructure,  and  data  (PAID).  At  any  particular  level  of 
interoperability,  a  set  of  specific  capabilities  must  be  present  for 
each  attribute  in  order  to  achieve  the  degree  of  interoperability 
maturity  defined  by  that  level. 

Capabilities  Model 

Defines  the  specific  capability  thresholds  (i.e.,  capability  suites 
across  PAID)  required  for  attaining  each  level  of  interoperability. 
This  model  provides  the  level  of  detail  needed  to  determine  systems 
interoperability  profiles  and  measures,  and  provides  the  basis  for 
conducting  LISI  assessments. 

Implementation  Options 
Tables 

Captures  the  full  range  of  possible  implementation  choices  that  are 
available  to  developers  for  implementing  each  of  the  capabilities 
identified  in  the  Capabilities  Model. 

Interoperability  Profiles 

The  interoperability  profile  for  a  particular  system  is  produced  as  a 
result  of  completing  the  LISI  questionnaire.  This  profile  contains 
the  specific  implementation  choices  made  by  a  particular  developer 
regarding  a  specific  system  or  application. 

Interoperability  Metrics 

Calculated  by  applying  the  Capabilities  Model  to  the  data  collected 
from  the  questionnaire.  Through  this  mapping,  a  profile  emerges 
which  depicts  the  organized  set  of  capabilities  exhibited  by  a 
system  in  terms  of  the  LISI  levels.  The  result  is  a  “measure”  which 
captures  the  level  of  interoperability  that  a  system  possesses. 

Comparison  Tables 

These  LISI  products  are  developed  by  comparing  and  assessing  the 

Architecture  Products 
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Capabilities  Model 


System  profile  and 


:  Generate  system 
profile  using  US! 
questionnaire 


DoD  Guidance 

'  Policy 

■  Dll -COE 
-  JTA 

■  Etc. 


[LISI  overlay  of  operational 
architecture 


Technology  Insertion 
Font  ms 


Other  system  profiles 
with  corresponding 
US!  metric 


— 

- 

A  1 

- 

1 

LISI  Data 
Repository 


Figure  4:  The  LISI  Interoperability  Assessment  Process 

Using  the  LISI  approach,  interoperability  measures  could  be  deduced  using  a  scorecard 
method,  as  shown  in  Figure  4  (adapted  from  a  1998  report  [C4ISR  98]).  In  the  absence  of 
precise  measures  and  recognizing  the  multidimensionality  of  interoperability,  it  is  reasonable 
to  use  scorecard  techniques  based  on  human  judgment  to  capture  how  well  a  unit  (or  the  DoD 
as  a  whole)  is  doing  with  respect  to 

•  technical  compliance 

•  system-to-system  interactions 

•  operational  mission  effectiveness 

5.1.1  Measuring  Technical  Compliance 

The  technical  view  of  an  architecture  focuses  on  the  criteria  governing  the  implementation  of 
specific  system  capabilities  or  attributes.  From  an  assessment  perspective,  the  concern  is 
whether  a  given  system's  implementation  complies  with  the  applicable  standards  and 
guidelines.  Therefore,  a  technical  scorecard  could  be  viewed  as  a  list  of  systems  with  ratings 
(pass/marginal/fail)  of  compliance  with  the  relevant  standards  and  guidelines. 

The  purpose  of  the  LISI  measure  is  to  capture  the  essence  of  potential  interactions  available 
between  systems,  as  registered  through  the  implementation  choices  made  by  developers.  The 
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measure  is  therefore  a  direct  reflection  of  the  comparison  of  interoperability  provided 
between  systems. 

The  LIS1  measure  provides  a  shorthand  definition  of  the  particular  form  of  interoperability  as 
expressed  in  the  LISI  maturity  model.  The  measure  comes  in  various  types  based  on  the 
nature,  purpose,  and  approach  to  performing  and  displaying  the  results  of  the  comparisons. 
An  example  of  the  various  options  for  describing  LISI  measures  is  shown  in  Table  7  (adapted 
from  a  1998  report  [C4ISR  98]). 


Table  8:  LISI  Interoperability  Measures 


Code 

Metric  type 

Generic 

G 

Expected 

E 

Specific 

S 

Level 

Enterprise 

4 

Domain 

3 

Functional 

2 

Connected 

1 

Isolated 

0 

Sublevel 

Varies  by  levels;  defined 
as  “a”  through  “z” 

a-z 

The  main  distinction  among  the  three  types  of  LISI  measures  is  the  comparison  of  a  single 
system  against  the  capabilities  model  (generic)  and  the  two  different  cases  where  two  or  more 
systems  are  compared  to  each  other  (expected  and  specific).  The  expected  level  of 
interoperability  between  two  systems  is  simply  the  lesser  of  the  two  systems’  generic  levels, 
or  the  level  at  which  one  would  expect  the  two  systems  to  interoperate.  The  specific  level  of 
interoperability  is  the  calculated  measure  between  two  systems  as  a  result  of  comparing  the 
specific  implementation  choices  that  each  system  has  made  regarding  the  registered  PAID 
capabilities.  The  specific  level  may  be  different  than  the  expected  level  based  on  the  added 
use  of  the  LISI  Options  Tables  and  the  consideration  of  the  technical  implementation  criteria. 
These  are  more  formally  defined  elsewhere  [C4ISR  98]. 

As  an  example,6  using  the  measures  in  Table  7,  consider  that  a  system  assessment  was 
conducted  and  the  LISI  measure  obtained  was  “G2c.”  Such  a  rating  of  the  inherent 
characteristics  of  this  system  would  mean  the  system  or  application  has  a  generic  level  of 
“2c.”  Therefore 

•  It  complies  with  JTA  and  DII-COE. 

•  It  can  operate  on  a  LAN. 

•  Its  environment  is  built  within  a  GUI. 


6  This  is  adapted  from  the  1998  report  [C4ISR  98], 
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•  It  supports  common  office  functions. 

•  Its  database  information  is  compliant  with  a  particular  functional  program. 

The  LISI  measure  obtained  from  these  comparisons  can  be  represented  in  several  formats, 
including  those  described  in  Table  8.  Figure  5  shows  the  example  populated  interoperability 
profile  for  this  system.  In  this  example,  the  system’s  generic  interoperability  level  is  2c,  the 
highest  level  at  which  a  capability  is  implemented  for  each  of  the  PAID  attributes. 


Table  9:  Possible  Formats  for  a  LISI  Measure 


Format 

Description 

Examples 

Summary  LISI  measure 

Only  the  major  level  and/or  sublevel  is  shown. 

G2,  E3,  G2b,  S3C 

Detailed  LISI  measure 

Individual  values  of  PAID  are  each  portrayed  as  separate 
components  within  the  measure 

G2(P3A2I3D2) 

Slb(P3a,  A2c,  I2b,  Dlb) 

Level 

Interoperability  Attributes 

(Environment) 

P 

A 

1 

D 

Enterprise 

C 

Level 

4 

b 

(Universal) 

a 

Domain 

c 

Service-approved 

TCP/IP  WAN, 

MIDB,  SQL 

Level 

Q 

b 

MNS  &  ORD, 

WAN  addressing 

NFS, 

SNMP, 

(Integrated) 

O 

a 

scheme 

ISDN  card 

Functional 

c 

IE  4.0 

NIFT,2 

Level 

(Distributed) 

2 

b 

Dll  COE  Compliant. 
Windows-std  file 
name  extensions 

MS  Office,  Access 
CMTK,  5D, 

IPLAN 

NES 

NTP.X.500 

USMTF, 

X.400, 

.wks,  .xls, 

DTED,  DBDB, 

a 

On  line 

Documentation 

Eudora 

TBS, 

LINK  16  &  22 

.ppt,  .doc, 

RPF,  CGM,  JBIG, 
JPEG,  HTML,  VPF 

Connected 

d 

Windows  Interface 

HF  Data  Modem, 

Level 

1 

c 

(JTA) 

FTP 

Kermit,  STU  III, 

MPEG  1.2 

(Peer-to-Peer) 

b 

ITU-T  Rec  X.509. 

Chat  2.0 

GSM  Cellular 

GKS,  wmf 

a 

Mil  Std  2045-28500 
Security  Labels 

Win32  API.PPS 

GBS 

Isolated 

d 

Login  procedures 

Level 

0 

c 

(Manual) 

b 

a 

0 

No  known  interoperability 

Level  2C 


Figure  5:  Example  Populated  Interoperability  Profile  for  2c  System 

In  addition  to  the  LISI  measure,  others  have  defined  architectural  attributes  that  could  serve 
as  indicators  of  interoperability.  These  appear  in  Appendix  A. 
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5.1.2  Potential  Systems  Interoperability  Scorecard 

The  systems  view  of  an  architecture  focuses  on  the  information  and  communications  systems 
that  are  brought  to  bear  to  support  the  information  flows  required  to  accomplish  operational 
missions.  The  Systems  Interoperability  Scorecard  attempts  to  measure  the  degree  to  which 
the  various  system  pairs  can  effectively  interoperate  in  context  to  meet  these  information 
flow  requirements. 

A  potential  interoperability  matrix  can  be  generated  for  a  group  of  systems  based  on  the 
generic  interoperability  level  of  each  system  and  the  specific  interoperability  level  for  each 
system  pair  within  the  group.  Figure  6  (adapted  from  Committee  [Committee  99])  presents  an 
example.  In  this  view,  a  scorecard  used  to  measure  interoperability  from  a  systems 
perspective  would  focus  on  the  ability  of  the  systems  in  each  pair  to  interact  with  one 
another.  The  scorecard  could  be  viewed  as  a  matrix  with  the  systems  represented  in  both  the 
rows  and  columns  and  entries  indicating  system-to-system  interoperability  as  inadequate 
(red),  marginal  (yellow),  or  adequate  (green). 


SI 

S2 

S3 

S4 

S5 

Sn 

SI 

S2 

G 

S3 

Y 

R 

S4 

Y 

G 

N/ 

S5 

G 

G 

R 

Y 

: 

■ 

■ 

■ 

■ 

s„ 

G 

Y 

G 

G 

Figure  6:  Example  Systems  Operability  Scorecard 

5.1.3  Measuring  Operational  Interoperability 

The  operational  view  of  an  architecture  addresses  particular-  mission  slices,  such  as  targeting, 
close  air  support,  and  force  sustainment  of  a  broader  operational  setting.  Within  each  slice, 
the  view  could  capture  the  players  involved  and  their  interactions,  their  functions,  decisions, 
actions,  and  the  flows  of  information  postulated  to  support  their  particular  roles  in  achieving 
overall  mission  effectiveness. 

The  review  of  a  system’s  Operational  Requirements  Document  will  determine  the  existence 
of  system  interoperability  requirements.  (Note:  system  interoperability  is  discussed  in  the 
next  section.)  The  first  step  in  measuring  compliance  of  these  requirements  is  to  trace  the 
requirements  through  the  system  functions.  This  may  be  accomplished  by  the  development  of 
operational  threads  (system  node  connectivity  or  link/node  diagrams)  or  paths  between  the 
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systems.  The  threads  are  identified,  traced,  and  developed  in  order  to  measure  and  quantify 
system  interoperability  [Leite  98]. 

A  scorecard  used  to  assess  interoperability  from  an  operational  architecture  perspective 
would  focus  on  the  ability  to  satisfy  specific  node-to-node  information  flow  requirements 
(that  describe  the  nature  of  the  information  and  services  needed,  its  directional  flow,  and  the 
constraints  and  demands  imposed  by  the  operational  environment.  This  is  illustrated  in  Figure 
7  (adapted  from  Committee  [Committee  99]).  The  assessed  degree  to  which  each  flow 
requirement  is  met  can  be  scored  using  green/yellow/red  ratings.  These  measures  are  often 
derived  from  lessons  learned  through  crises  or  exercises  including  observed  events  and 
anecdotal  feedback  [Committee  99] . 


Mission  Slice  #3 


Mission  Slice  #2 


Figure  7:  Example  of  an  Operational  Interoperability  Scorecard 

At  this  level  it  is  possible  to  estimate  and  measure  important  quality  attributes  associated  with 
interoperability.  Leite  has  defined  these  relationships  and  they  are  summarized  in  Table  9 
[Leite  98].  The  mathematical  relationships  for  each  of  these  measures  are  defined  in 
Appendix  A. 
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Table  1 0:  Quality  Attributes  Associated  with  Interoperability 


Attribute  Measure 

Description 

Connectivity 

Connectivity  can  be  directly  measured  by  counting  the  number  of  messages 
initiated  by  all  participating  units  and  the  number  of  messages  received  for  the 
network  or  data  link.  To  the  extent  that  the  link  is  in  continuous  operation,  the 
connectivity  sampled  in  this  manner  is  representative  of  network  connectivity. 

If  the  network  is  operated  intermittently,  then  the  sample  must  be  carefully 
selected  and  tested  to  ensure  that  the  required  confidence  level  is  attained. 

Capacity 

The  capacity  of  a  system  is  the  rate  at  which  data  may  be  passed  over  time. 
Given  its  operating  parameters,  a  maximum  data  rate  can  be  calculated  for  any 
system  or  group  of  systems. 

System  overload 

A  system  overload  occurs  when  more  data  must  be  exchanged  than  the  system 
is  able  to  transmit.  Typically,  the  overload  is  placed  in  a  queue  and  is  then 
transmitted  when  capacity  is  available.  Therefore,  the  measure  of  system 
overload  is  the  sum  of  the  messages  remaining  in  queues  after  their  assigned 
transmission  period  for  all  system  nodes. 

Underutilization 

Underutilization  occurs  when  the  system  data  rate/message  load  is  less  than  its 
full  capacity  but  messages  are  waiting  in  queues  to  be  transmitted.  This  occurs 
when  the  item  slot  or  transmission  allocation  to  selected  nodes  is  less  than  that 
required  to  clear  the  queue  by  the  end  of  a  transmission  period.  Similarly  other 
nodes  do  not  use  all  of  their  allocated  time. 

Undercapacity 

Undercapacity  occurs  when  messages  remain  in  queues  and  the  system  data 
rate  is  at  the  maximum. 

Data  latency 

Data  latency  is  the  elapsed  time  from  the  time  of  the  event  to  the  time  of 
receipt  by  the  user  (tactical  data  processor).  For  analytical  puiposes,  the 
latency  is  often  divided  into  smaller  segments.  Several  common  time  periods 
are  the  following: 

•  time  of  event  to  time  of  observation 

•  time  of  observation  to  completion  of  processing 

•  completion  of  processing  to  time  of  receipt  at  the  tactical  data 
processor 

This  division  is  useful  in  situations  involving  a  remote  sensor  and  intermediate 
processing  to  reduce  the  data  to  a  usable  form  (track  message)  prior  to  passing 
the  data  to  the  user. 

Information  interpretation 
and  utilization 

Having  passed  the  data  and  correctly  interpreted  it,  the  next  step  would  be  to 
verify  that  the  proper  action  is  taken.  Verification  of  the  action  taken  involves 
a  review  of  the  logic  associated  with  each  option  that  is  possible  in  response  to 
a  message  or  operator  action.  These  deal,  of  course,  with  questions  of 
interoperability  and  not  with  the  difficult,  higher  level  topic  of  measuring 
mission  effectiveness. 
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5.2  Management  Measures  Associated  With  Interoperability 

The  magnitude  of  the  technology-driven  transformation  required  to  achieve  interoperability 
points  to  the  enormity  of  the  institutional  challenges  associated  with  the  transformation.  The 
Committee  to  Review  DoD  C4I  Plans  and  Programs  found  that  “achieving  C41 
interoperability  is  more  a  matter  of  organizational  commitment  and  management7  than  one  of 
technology”  [Committee  99] . 

Adequate  C4I  interoperability  is  inherently  a  distributed,  horizontal  challenge  that  must  be 
addressed  in  a  largely  vertical  world.  Enabling  fast  and  effective  responses  to  this  challenge 
requires  that  interoperability  be  built  into  the  force  structure  across  service  and  unit 
boundaries.  This  means  that  there  must  be  incentives  for  investments  and  actions  across 
organizational  boundaries.  Crossing  these  boundaries  is  particularly  important  to  the 
development  and  fielding  of  systems  that  support  joint  operations.  The  DoD  must  search  for 
practical  ways  to  reward  interoperability. 

Measures  are  important  to  senior  decision  makers.  The  Information  Technology  Reform  Act 
(ITMRA)  of  1996  (Public  Law  104-106),  also  known  as  the  Clinger-Cohen  Act,  requires  the 
Federal  government  to  develop 

“. .  .a  process  and  procedure  for  establishing  goals  for  improving  the 
efficiency  and  effectiveness  of  government  agencies  operations  and  the 
ability  to  deliver  goods  and  services  to  the  public  using  Information 
Technology.  The  goals  must  be  measurable.”8 

Achieving  large-scale  cultural  change  in  an  organization  to  achieve  C4I  interoperability 
requires  commensurate  change  in  management  and  the  organizational  measures.  In  large 
organizations,  the  behavior  of  personnel  is  strongly  influenced  by  the  measures  that 
management  uses  to  assess  performance,  whether  those  measures  are  part  of  a  formal 
assessment  or  are  more  perceived  than  formal.  People  are  keenly  aware  of  what  matters  in 
terms  of  rewards,  promotion,  and  credit  and  they  behave  in  a  manner  consistent  with  their 
perceptions.  Good  management  measures  help  to  drive  organizational  behavior  that  supports 
areas  of  operational  significance.  In  general,  management  measures  focus  on  organizational 
performance  or  characteristics  and  are  used  by  senior  management  to  assess  the  effectiveness 
of  the  organization  and  its  leadership. 

The  Committee  to  Review  DoD  C4I  Plans  and  Programs  has  identified  the  following  list  of 
possible  management  measures  to  promote  interoperability  across  the  board: 

•  Number  of  C4I  systems  that  conform  to  the  Joint  Technical  Architecture 


7  This  includes  allocation  of  resources,  attention  to  detail,  and  continuing  diligence. 
s  Please  see  http://wwwoirm.nih.gov/policy/itmra.html  for  more  information. 
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•  Number  of  individuals  trained  in  the  use  of  specific  C4I  systems 

•  Number  of  C4I  systems  “certified”  to  be  interoperable 

•  Time  or  personnel  required  to  develop  time -phased  force  and  deployment  data  or  an  air- 
tasking  order 

•  Time  needed  to  stand  up  a  tactical  network  for  a  joint  task  force 


5.3  Summary  of  Recommended  Measures 

The  Practical  Software  and  Systems  Measurement  (PSM)  approach  is  based  on  actual 
measurement  experience  on  government  and  industry  projects  [PSM  98].  It  represents  a 
collection  of  best  practices  used  by  measurement  professionals  within  the  software  and 
systems  acquisition  and  engineering  communities.  The  PSM  selection  and  specification 
approach  is  based  on  the  direct  relationship  between  project  issues,  information  needs,  and 
the  measures  that  support  the  required  information.  To  implement  this  approach,  PSM  maps 
Common  Issue  Areas  to  related  Measurement  Categories,  and  then  to  measures  in  each 
category.  The  following  table  summarizes  recommended  measures  that  have  been  organized 
into  the  PSM  Issue-Category-Measure  (ICM)  structure. 


Table  1 1:  Summary  of  Recommended  Measures 


Specific  Issues 

Common  Issue  Area 

Measurement  Category 

Recommended  Measure 

Compliance 

With  Standards 

Technical  Adequacy 

Technical  Performance 

LISI  generic  level  of  interoperability 

Systems 

Interoperability 

Technical  Adequacy 

Technical  Performance 

LISI  expected  level  of  interoperability 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

LISI  specific  level  of  interoperability 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Connectivityt 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Capacity f 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

System  Overload1^ 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Underutilization 1 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Undercapacity f 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Data  latency^ 

Operational 

Interoperability 

Technical  Adequacy 

Technical  Performance 

Information  interpretation  and 
utilization 

^  For  a  formal  definition  of  this  measure,  refer  to  Appendix  D. 
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Management 

Commitment 

Schedule  and  Progress 

Milestone  Performance 

Number  of  C4I  systems  that  conform 
to  the  Joint  Technical  Architecture 

Management 

Commitment 

Schedule  and  Progress 

Milestone  Performance 

Number  of  C4I  systems  “certified”  to 
be  interoperable 

Management 

Commitment 

Development 

Performance 

Productivity 

Time  needed  to  stand  up  a  tactical 
network  for  a  joint  task  force 

Management 

Commitment 

Development 

Performance 

Productivity 

Time  or  personnel  required  to  develop 
time-phased  force  and  deployment  data 
or  an  air-tasking  order 

Management 

Commitment 

Resources  and  Cost 

Personnel 

Number  of  individuals  trained  in  the 
use  of  specific  C4I  systems 

5.2  Tradeoff  Analysis 

Although  interoperability  is  a  critical  enabler  for  military  operations,  it  must  be  recognized  as 
just  one  of  several  technical  attributes  of  any  system  of  systems.  Military  commanders  need 
many  things  from  their  C4I  systems  besides  interoperability,  and  tradeoffs  among  these  needs 
are  often  required.  Other  attributes  will  sometimes  be  in  competition  with  interoperability 
and  with  each  other.  In  thinking  about  overall  system  functionality  or  performance,  security 
requirements  such  as  confidentiality,  authentication,  non-repudiation,  integrity,  and  system 
availability  must  be  considered  together  with  interoperability.  An  appropriate  balance  must  be 
sought.  For  example,  there  are  tradeoffs  between  security  and  interoperability. 

Interoperability  can  promote  an  attacker’s  access  to  diverse  systems,  thus  facilitating  the 
rapid  spread  of  attacks. 


5.3  Recommendations 

The  following  recommendations  are  made  for  further  exploration: 

•  Investigate  current  efforts  to  track  interoperability  on  a  comprehensive  basis.9  This 
includes  investigating  the  maturity  and  use  of  LISI  as  a  framework  to  assess 
interoperability. 10 

•  Analyze  results  from  well-instrumented  simulations  and  exercises  to  evaluate  tradeoffs 
between  interoperability  and  other  fundamental  attributes  of  C4I  systems,  including 
security,  availability,  flexibility,  survivability,  and  performance. 

•  Examine  scenario-based  assessment  and  architectural-style-based  assessment  as  a  way  to 
better  understand  interoperability  measures  and  the  tradeoffs  involved  between  other 


9 

The  Committee  to  Review  DoD  C4I  Plans  and  Programs  determined  that  despite  laudable  case-by-case  efforts 
to  track  interoperability,  there  is  today  no  method  for  tracking  interoperability  on  a  comprehensive  or 
systematic  basis  [Committee  99] . 

10  CJCSI  6212. OIC.  November  20,  2003,  Mandate  LISI  assessments  for  all  ICIDS  Acquisition  Category 
programs  referenced  in  CJCSI  3170.012,  CJCSM3170.OIM  and  all  not-ACAT  and  fielded  systems. 
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quality  attributes  of  a  system. 1 1  Investigate  appropriate  interoperability  measures  using 
the  Architecture  Tradeoff  Analysis  Method  (ATAM)  [Kazman  00]. 

•  Explore  the  use  of  multivariate  analysis  to  take  into  account  the  likely  interdependence  of 
various  interoperability  measures  and  competing  system  quality  attributes. 


Investigations  in  this  area  would  be  based  on  foundational  work  described  by  Bass  et  at.,  Barbacci  et  al.,  and 
Taylor  [Bass  98],  [Barbacci  95],  [Taylor  00]. 
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Appendix  A 


Some  Historical  Definitions  of  Interoperability 


A  number  of  reports  and  technical  papers  have  defined  interoperability. 

•  “The  effort  required  to  couple  one  system  with  another”  [McCall  80]. 

•  “The  ability  of  systems,  units,  or  forces  to  provide  services  to  and  accept  services  from 
other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged  to  enable  them  to 
operate  effectively  together”  [DoDD  77]. 

•  “The  ability  of  one  services’  system  to  receive  and  process  intelligible  information  of 
mutual  interest  transmitted  by  another  service’s  system”  [JINTACCS  74], 

•  “The  ability  of  one  system  to  receive  and  process  intelligible  information  of  mutual 
interest  transmitted  by  another  system”  [Eldridge  78]. 

•  “The  ability  of  two  or  more  systems  or  components  to  exchange  information  and  to  use 
the  information  that  has  been  exchanged”  [IEEE  90] . 

There  are  problems  with  McCall’s  definition.  Here,  “coupling”  includes  linking  two 
programs  to  interoperate  on  a  single  computer  or  linking  programs  on  separate  computers  to 
interoperate.  The  quality  factor,  interoperability,  is  therefore  important  when: 

•  retrofitting  two  or  more  previously  developed  systems  into  one  system 

•  developing  new  systems  independently  that  will  interoperate  with  each  other 

•  developing  a  system  with  the  expectation  that  it  will  eventually  interoperate  with  an,  as 
yet,  undefined  future  system 

The  definition,  taken  literally,  includes  only  the  connectivity  and  compatibility  issues  of 
interoperability,  implying  only  equipment-level  considerations.  But  hardware  compatibility 
does  not  assure  interoperability.  For  example,  two  persons  using  exactly  the  same 
transmitters  on  the  same  frequency  may  not  be  able  to  interoperate,  particularly  if  one  person 
speaks  only  English,  the  other  speaks  only  Arabic.  Interoperability  is  achieved  when  both 
persons  can  transmit  and  receive  information  of  mutual  use  and  understandability. 

Eldridge’s  definition  of  interoperability  is  also  unsatisfactory  because  it  stresses 
standardization.  The  emphasis  on  standardization  of  hardware  and  software  overlooks  the 
content  of  the  messages  and  the  differing  operational  requirements  that  affect  interoperability. 

The  JINTACCS  and  DoDD  2010.6  [DoDD  77]  definitions  are  preferable.  These  definitions 
seem  to  most  accurately  define  the  ultimate  meaning  of  interoperability — as  a  broad  and 
complex  subject  rather  than  a  binary  attribute  of  systems. 
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More  recently,  the  Joint  Chiefs  of  Staff  Publication  1-02  [DoD  98]  defines  interoperability  in 
a  way  that  acknowledges  the  technical  and  operational  components  that  contribute  to  a  more 
meaningful  interpretation. 
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Appendix  B 


Testing  Interoperability 


C4I  interoperability  depends  on  architecture  and  the  resultant  requirements  specification. 
Testing  compares  actual  performance  with  requirements.  Ensuring  that  the  architecture  and 
requirements  are  successfully  implemented,  and  that  the  required  level  of  interoperability  is 
achieved,  requires  comprehensive  testing  and  evaluation. 

Testing  can  take  place  in  a  laboratory,  field  location,  or  at  an  individual’s  workstation  (during 
early  systems  designs).  Typically,  systems  are  tested  at  different  stages  in  their  life  cycle, 
during  development,  preproduction,  and  in  the  field. 


Developmental  Assesses  progress  in  meeting  system-level  requirements  ranging  from  functionality  to 
testing  performance.  To  ensure  correct  intent,  a  system’s  “paper”  requirements  may  be  tested 

against  user-stated  needs. 


Preproduction  Conformance  testing  focuses  on  the  stand-alone  functionality  and  performance  of  a 
testing  particular  system  in  terms  of  stated  requirements  (through  a  paper  or  laboratory  test). 


System-to-system  testing  determines  how  well  a  system  interoperates  with  other 
systems.  It  is  typically  performed  in  a  laboratory  where  two  or  more  systems  can  be 
interconnected.  Its  scope  can  range  from  “lower-layer”  (e.g.,  communications)  to 
“higher  layer”  (e.g.,  applications  and  data)  interoperability. 


Field  testing 


Assesses  the  extent  to  which  a  system  satisfies  users’  operational  needs  in  a  “real- 
world”  setting. 


Functional  testing  involves  configuring  systems  to  meet  the  unique  demands  of 
particular  customers,  integrating  products  with  the  embedded  base  of  systems,  and 
evaluating  the  resulting  system  of  systems  from  the  end-to-end  functional  perspective. 


Follow-on  testing  assesses  a  system’s  performance  after  it  has  been  fielded,  re¬ 
verifying  interoperability  periodically  or  as  changes  occur  and  providing  a  mechanism 
for  tracking  progress  in  addressing  known  problems. 


Leite  proposes  the  test  assessment  method  summarized  in  Figure  8  [Feite  98].  Testing  should 
be  seen  as  an  integral  part  of  requirements  definition  and  system  development.  Thus  testing 
must  be  essentially  continuous,  and  “stability”  is  a  state  that  is  never  reached  in  any 
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meaningful  sense.  Without  ongoing  feedback,  initial  implementations  of  processes  and 
systems  may  interoperate  satisfactorily  at  first,  but  not  later  [Committee  99]. 


Figure  8:  Interoperability  Assessment  Process 


Often,  requirements  are  strong  in  specifying  behavior  under  ideal  conditions  but  weak  in 
defining  what  should  happen  under  adverse  situations  (e.g.,  response  of  a  system  to  failure). 


Many  interoperability  problems  are  subtle  and  manifest  themselves  only  in  certain  sets  of 
circumstances.  They  are  often  hard  to  uncover  and  demand  a  great  deal  of  empirical  work 
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and  testing  to  resolve.  Research  on  the  theory  and  practice  of  interoperability  testing  has 
begun  only  very  recently  [Kang  98]. 
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Appendix  C 


Potential  Measures  of  Interoperability 


Tables  1 1  and  12  present  potential  measures  (proposed  by  other  authors)  that  may  provide 
insight  into  interoperability.  Table  12  is  adapted  from  Kalyanasundaram  [Kalyanasundaram 
98]. 

Table  1 1:  Software  Interoperability  Categorical  Measures 


Category 

Measures 

Standards 

standards  explicitness 

standards  maturity 

standards  vendors  supporting 

standards  feature  coverage 

standards  sufficiency 

Profiles 

profile  explicitness 

profile  width 

profile  coverage 

profile  extensions 

use  during  product  selection 

profile  sufficiency 

profile  documentation 

Products 

products  available  supporting 

product  performance 

platforms  supported 

Conformance 

degree  of  conformance 

product-to-product  interoperability 
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Table  12:  Architecture  Measures  Related  to  Interoperability 


Measure 

Code 

Description 

Number  of  3rd  party  components 

No3C 

The  count  of  the  number  of  components  that  have  been  acquired  from  outside 
vendors.  This  measure  is  a  reflection  of  the  “openness”  of  the  architecture. 

Number  of  components 

NoC 

The  total  number  of  architectural  units  that  can  be  found  in  the  system  of 
interest.  (A  measure  of  the  size  of  the  system.) 

Number  of  control  components 

NoCC 

Number  of  total  components  that  provide  logical  operations  on  a  given  set  of 
input  data.  An  example  of  control  components  is  a  software  structure  that  acts  as 
an  iterator.  This  number  is  a  subset  of  NoC. 

Number  of  data  components 

NoDC 

Total  number  of  components  that  are  passive  in  nature.  Examples  include 
databases,  stacks,  and  shared  memory  units. 

Total  number  of  external 
interfaces 

NoDC 

Number  of  connectors  that  allow  the  component  to  interact  with  other 
components  outside  the  system  or  subsystem.  This  is  an  indirect  measure  that 
tries  to  capture  the  coupling  in  the  system. 

Total  number  of  internal 
interfaces 

TNII 

The  number  of  connectors  that  allow  components  to  interact  with  other 
components  within  a  subsystem  or  layer. 

Number  of  specialized 
components 

NoSC 

A  count  of  components  that  are  considered  to  have  a  high  level  of  system 
specificity.  These  components  offer  specific  services,  which  prohibit  their  use  in 
any  other  context.  This  measure  is  based  on  a  subjective  measure  provided  by 
the  architects  and  designers. 

Number  of  functionally  critical 
components 

NFCC 

Counts  the  number  of  components  whose  failure  would  drastically  affect  the 
systems’  functionality.  This  is  a  subjective  measure  provided  by  the  designers. 

Number  of  shared  memory 
components 

NSMC 

Shared  memory  is  a  critical  component  in  large-scale  systems  with  a  high  level 
of  criticality  associated  with  it. 

Number  of  architectural 
revisions 

NoAR 

Represents  the  number  of  changes  that  the  architecture  has  gone  though  before 
reaching  the  current  productization  level. 

Number  of  interface  types 

NoIT 

In  large  systems,  several  types  of  interactions  are  available.  This  is  a  measure  of 
the  various  interface  techniques  in  the  system.  The  more  the  NoIT,  the  higher 
the  complexity. 

Number  of  generic  components 

NoGC 

This  is  a  measure  of  components,  which  are  “general”  in  nature.  These 
components  are  not  domain  specific.  (The  designer  would  label  each  component 
as  generic  or  specialized.) 

Number  of  redundant 
components 

NoRC 

In  some  systems,  hardware  components  and  software  components  are  replicated 
to  recover  during  component  failures.  Redundant  components  are  components 
that  are  generally  not  used  by  the  system  during  normal  operation  and  which 
usually  mirror  the  functionality  of  other  components  in  the  system  that  are  used 
extensively.  This  measure  is  provided  by  the  system  documentation. 

38 


CMU/SEI-2004-TN-003 


Number  of  concurrent 
components 

NCC 

The  number  of  components  that  operate  concurrently.  Concurrency  is  prevalent 
in  real-time  telecommunications  systems.  Its  effect  on  the  quality  of  the  system 
is  significant. 

Number  of  subsystems 

NoSS 

Represents  the  number  of  units  that  are  logical  or  physical  clusters  or 
components. 

Number  of  services 

NoS 

The  number  of  different  services  that  are  offered  by  a  telecommunications 
system. 
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Appendix  D 


Equations  for  Quality  Attributes  Associated 
with  the  Interoperability  Scorecard 


Connectivity 

A  connectivity  index  can  be  calculated  for  any  communications  system.  It  is  a  relationship 
between  the  number  of  system  nodes  and  the  available  paths.  The  connectivity  index  is 
defined  by  the  equation: 


n  *  (n  - 1) 


Where: 

Cj=  Connectivity  index 

k  =  Number  of  connections  (paths  between  nodes) 
n  =  Number  of  nodes  (participating  units) 


Connectivity  can  be  measured  directly  by  counting  the  number  of  messages  initiated  by 
all  participating  units  and  the  number  of  messages  received  for  the  network  or  data  link. 
To  the  extent  that  the  link  is  in  continuous  operation,  the  connectivity  sampled  in  this 
manner  is  representative  of  network  connectivity.  If  the  network  is  operated  intermittently, 
then  the  sample  must  be  carefully  selected  and  tested  to  ensure  that  the  required 
confidence  level  is  attained. 

The  general  relationship  for  measuring  the  connectivity  is  the  following: 


n 


x=l 


Where: 

C  =  Node  Connectivity  (during  measurement  period) 
nr  =  Number  of  receiving  nodes 
n,  =  Number  of  transmitting  nodes 
M,  =  Messages  transmitted  by  a  node 
Mr  =  Messages  received  by  a  node 

Information  Flow 

The  volume  of  data  is  typically  a  function  of  the  tempo  of  operations  and  the  area  of 
interest.  The  area  of  interest  (AOI)  is  defined  by  the  operational  commander.  The  tempo 
of  operation  is  event-driven;  however,  estimates  are  possible  based  on  historical  and 
exercise  results. 

Capacity  is  a  function  of  the  available  data  links.  In  practice,  multiple  links  or  paths  are 
available.  For  weapon  and  combat  systems,  there  is  a  requirement  for  primary  and  back- 
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up  paths.  The  redundancy  of  data  flow  limits  the  total  capacity  to  an  amount  that  is  less 
than  the  sum  of  the  individual  systems. 

Several  items  may  be  measured  or  calculated  with  respect  to  system  performance.  They 
are  capacity,  system  overload,  and  data  latency.  The  relationships  for  these  measures 
follow: 


❖  Capacity 

The  capacity  of  a  system  is  the  rate  at  which  data  may  be  passed  over  time.  Given 
its  operating  parameters,  a  maximum  data  rate  can  be  calculated  for  any  system  or 
group  of  systems.  These  relationships  are  described  as  follows: 

Qeff  ~  (Gmax  ~  Qoh  )  X  Qf  _  ^ p  ) 

Where: 

Qeff  =  Effective  system  capacity  (data  rate) 

Qmax  =  Maximum  data  rate 

Q0h  =  System  overhead  data  rate 

tf  =  Time  slot  duration  (unit  transmission) 

tp  =  Unit  propagation  time 

❖  System  Overload 

A  system  overload  occurs  when  more  data  must  be  exchanged  than  the  system  is 
able  to  transmit.  Typically,  the  overload  is  placed  in  a  queue  and  is  then  transmitted 
when  capacity  is  available.  Therefore,  the  measure  of  system  overload  is  the  sum  of 
the  messages  remaining  in  queues  after  their  assigned  transmission  period  for  all 
system  nodes. 


Mol  =ntx 


y=i 


Where: 

Mol  =  System  message  overload 

n,  =  Number  of  transmitting  nodes 

Mq  =  Messages  in  queue  to  be  transmitted  by  node 

❖  Underutilization 


This  occurs  when  the  system  data  rate/message  load  is  less  than  its  full  capacity  but 
messages  are  waiting  in  queues  to  be  transmitted.  This  occurs  when  the  item  slot  or 
transmission  allocation  to  selected  nodes  is  less  than  that  required  to  clear  the  queue 
by  the  end  of  a  transmission  period.  Similarly  other  nodes  do  not  use  all  of  their 
allocated  time. 

Quu  =  M0l 


For  Mol  <  {Qeff  -  Q) 

AND 

Quu  =  Qeff  ~  Q 

For  Mol  >  ( Qeff  -  Q) 


Where: 

Quu  =  System  Underutilization  (data  rate) 
Q  =  Measured/observed  data  rate 

(Other  terms  as  previously  defined.) 
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❖  Undercapacity 

Undercapacity  occurs  when  messages  remain  in  queues  and  the  system  data  rate  is 
at  the  maximum. 

Qyc  =  (Q  +  M0l)~  Qeff 

Must  be  >  0 

Where: 

Quc  =  System  undercapacity  (data  rate) 

(Other  terms  as  previously  defined.) 

Data  Latency 


Data  latency  is  the  elapsed  time  from  the  time  of  the  event  to  the  time  of  receipt  by 
the  user  (tactical  data  processor).  For  analytical  purposes,  the  latency  is  often 
divided  into  smaller  segments.  Several  common  time  periods  are  the  following: 

•  time  of  event  to  time  of  observation 

•  time  of  observation  to  completion  of  processing 

•  completion  of  processing  to  time  of  receipt  at  the  tactical  data  processor 

This  division  is  useful  in  situations  involving  a  remote  sensor  and  intermediate 
processing  to  reduce  the  data  to  a  usable  form  (track  message)  prior  to  passing  the 
data  to  the  user.  These  relationships  are  expressed  as  follows. 


A  t  =  tr-  te 
A ~to=t0-  te 
A t  m  =  tm  ~  t0 

A ~tr=tr-  tm 


Equation  8  may  be  rewritten  as: 

At  =  ~Kt~0  +  ~Kt~^  +  ~KFr 

Where: 

At  =  Time  latency 

A  t0  =  Latency  of  observation 

A  tm  =  Latency  of  measurement/processing 

A  tr  =  Latency  of  transmission/receipt 

te  =  Time  of  event 

t0  =  Time  of  observation 

tm  =  Time  of  completion  of  processing 

tr  =  Time  of  receipt 
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Information  Interpretation  and  Utilization 

Having  passed  the  data  and  correctly  interpreted  it,  the  next  step  would  be  to  verify 
that  the  proper  action  is  taken.  Verification  of  the  action  taken  involves  a  review  of 
the  logic  associated  with  every  option  that  is  possible  in  response  to  a  message  or 
operator  action.  These  deal  with  questions  of  interoperability  and  not  with  the  difficult, 
higher-level  topic  of  measuring  mission  effectiveness.  These  data  would  be 
qualitative  in  nature,  perhaps  binary  (i.e.,  successful  vs.  failed).  Some  suggested 
measures  in  this  area  include 

•  Percentage  of  initial  transmission  messages  received  correctly  by  shooters 

•  Percentage  of  consistency/disparity  of  redundant  data  sources 

•  Number  of  tries  needed  to  establish  connections 

•  Delay  in  sending  critical  command  messages  and  time  to  receive  and 
acknowledge  messages 
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